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(57) Abstract 

Methods for producing and multiplexing a file format, as well as structures for a hierarchical file format and data file are provided. 
The data file may include data that is divided in a hierarchical manner, including a highest level document portion (202) that supports all 
lower level portions of the data file. The hierarchical data file forms a multimedia document that can be displayed on a computer display 
with accompanying audio. The multimedia document may include data in a variety of file formats, including image data (214), sound data 
(217), textual data (218) and video data. At least some of the data is preferably multiplexed in the data file so that the multiplexed data 
can be progressively played and displayed as it is downloaded by the computer. Data that can not be progressively played need not be 
multiplexed in the data file and can be located in an area of the data file separate from the multiplexed data. 
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ENCAPSULATED DOCUMENT AND FORMAT SYSTEM 
BACKGROUND OF THE INVENTION 

Field of the Invention 

This invention relates to producing a data file 
5 having different file format elements encapsulated within 
the data file. More particularly, the present invention 
relates to a data file in which different types of data, 
such as video and sound, are stored in a unified 
encapsulated format that can be repeatedly assembled, 
10 stored, and sent. 

Description of the Related Art 

Modern computer files may include parts 
representing various kinds of information. The parts are 
specially coded to represent the information. Moreover, 

15 each set of bits in modern computer files is coded in a 
unique way to represent a unique kind of information. For 
example, the ASCII coding system assigns a bit sequence 
to each alphanumeric character of the ASCII set. That bit 
sequence represents the character, however, only if one 

20 knows that it is ASCII. 

Bits can also be used to represent image and sound 
data. For example, image- data may be represented in 
BITMAP format, or in various compressed image data 
formats. The set of bits itself, however, has no meaning 

25 unless one knows the format. The ASCII code for a 

particular alphanumeric character could equally well 
represent a sequence of sound, or a portion of an image. 
Different kinds of files are used, therefore, to transfer 
different kinds of information. 
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It has become highly desirable to transmit 
different kinds of information, such as graphics, video, 
and/or sound, over a channel to a user. A so-called 
multimedia document includes different kinds of 
5 information. One example of a multimedia document is 

found in hypertext markup language (HTML) encoded format 
on the worldwide web. This information is a document in 
the sense that, once downloaded, a document-like image 
appears on the computer display. (As used in the 

10 remainder of the description, the term document is to be 
construed broadly to incorporate data files that include 
text, images, and/or other information that can be 
presented on a computer like a printed document.) The 
display shows rich text (i.e., text that is coded in 

15 terms of its size, color, and font) as well as images 
located at different locations within the document. A 
worldwide web downloaded document can also include sound. 

The various kinds of information in a multimedia 
document are each formed from a separate file that 

20 includes instructions about where the pieces of the 
document should go. For example, an HTML document 
includes rich text information, along with commands to 
insert additional information. Such additional 
information, typically GIF or JPEG image files, is 

25 separately downloaded and inserted into the text of the 
document . 

The original format for downloading HT ML documents 
displayed the formed document only when it was completely 
downloaded. While modern enhancements allow portions of 
30 the image to be displayed while downloading, most do not 
allow playing of the sounds added to the download until 
the sound file is completely received. 
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The download of HTML documents cannot be sequenced 
or choreographed, because information is downloaded as it 
becomes available. For example, if the document includes 
multiple images, they may come in any order. Moreover, 
5 actions may not be choreographed with image download or 
with text download. 

It is difficult to save an HTML document, because 
the document is a text document that includes commands, 
for additional images. The images are not part of the 

10 HTML document, but rather separate sub-documents. The 

perceived display of an HTML documents is thus assembled 
from separate downloaded pieces of information. HTML 
documents cannot be easily saved, however, because no 
connection exists between the various pieces of 

15 information and the various sub-documents. Moreover, HTML 
documents cannot be easily reassembled or resent. While 
proposals do exist for "encapsulated" HTML that would 
allow an HTML document and its component parts to be 
concurrently saved for later viewing or forwarding, these 

20 proposals suffer from a variety of restrictions, and none 
would allow for data to be choreographed by the author. 
For the remainder of this description, choreographing 
will refer to allowing a document author to select or 
define the relative times at which various document 

25 components, such as images, text, or sound, are played or 
displayed by a computer. 

SUMMARY OF THE INVENTION 

The present invention provides a system and method 
for forming document files as well as a file format, each 
30 of which can include integrated information. The 

information is stored in a special form that allows parts 
to be viewed while forming the file. 
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Preferably, all of the information, of all of the 
different kinds, is stored in the file in a unified, 
encapsulated format. In an encapsulated document, 
therefore, the data defining the document is integrated 
5 into a single file containing the data and having an 
overall format defining the document. As a result, the 
file can be repeatedly stored and sent, because the 
encapsulated format facilitates keeping the file 
structure intact after the user has viewed the file. 

10 Alternatively, only the formatting data may be 

contained in the document encapsulation (i.e., the size, 
position, and other display characteristics of the 
document, irrespective of the actual document data). The 
actual document data could then be stored in and 

15 retrieved from an external source, such as an URL, an 
HTML document, or a cache, or from another source, such 
as a CD. The user can still save and resend the 
encapsulated document, without the problems of HTML, as 
the document data can be retrieved from the external 

20 sources and encapsulated into the document. 

The invention can multiplex different forms of 
different data together. The multiplexed data is in a 
streaming format, also known as a progressive rendering 
format, especially when dealing with image formats. The 

25 streaming format data, once downloaded, causes the 
display to change on an incremental basis. Further 
downloading occurs simultaneously with the displaying of 
the file. 

Certain kinds of data images may be in a non- . 
30 streamable format. For example, images that are not 

susceptible of progressive rendering cannot be sent in a 
form that facilitates updating user perception on an 
incremental basis. The preferred mode of the invention, 
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however, updates user perception of the displayed data 
whenever available . 

It is another object of the present invention to 
provide a system that permits a number of different file 
5 formats to be encapsulated in a way that enables 
choreographing between the file elements. 

Special techniques of the present invention are 
used with so-called temporal data. In the present 
specification, temporal data includes data that must be 

10 presented to the user during a specific time span, for 
example, sound files. Once a sound file begins playing, 
it should be played uninterrupted to avoid an unnatural 
feel to the sounds. The temporal operations of the 
present invention are used as part of the choreography. 

15 This encapsulated data format is especially advantageous 
when combined with the special modules of the present 
invention that are described in this specification. The 
present invention recognizes temporal data objects and 
creates a multiplexed format in which temporal 

20 presentations are accurately conveyed to the user. 

In a first embodiment, the invention is a method 
for producing a hierarchical data file for a multimedia 
document. The data file has different file formats 
encapsulated within the data file. The method includes 

25 several steps: (a) encapsulating in a multimedia document 
a first file support object including information in a 
first file format; (b) supporting the first file support 
object by the multimedia document; © encapsulating in the 
multimedia document a second file support object 

30 including information in a second file format different 
from the first file format; and (d) supporting the second 
file support object by the multimedia document. 
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In another embodiment, the invention is a 
hierarchical data file structure that encapsulates 
different file formats to form a multimedia document. The 
multimedia document is capable of being displayed on a 
5 display of a computer system. The data file includes a 
document that includes information for controlling the 
display. The data file also includes a first support 
object including information in a- first file format. The 
first support object is encapsulated in the document and 

10 is capable of supporting first lower objects. Each first 
lower object is a lower level object than the first 
support object in the hierarchical data file structure. 
The data file also includes a second support object 
including information in a second file format different 

15 from the first file format. The second support object is 
also encapsulated in the document and is capable of 
supporting second lower objects. Each second lower object 
is a lower level object than the second support object in 
the hierarchical data file structure. 

20 Instill another embodiment , the invention is a 

method for encoding a framed image in a frame to be 
included as part of a multimedia document. The 
multimedia document encapsulates data in different file 
formats and is capable of being displayed on a display of 

25 a computer, which includes a player. The method 

comprises: (a) placing an image into the multimedia 
document; (b) receiving information about the image by 
the player; © decoding the image information; <d) sending 
the decoded image information to the multimedia document; 

30 and (e) enclosing the decoded image in a frame in the 
multimedia document . 

In another embodiment , the present invention is a 
method for multiplexing data in a multiplex message that 
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includes data in different file formats. The file formats 
are selected from a group of file formats including a 
textual format, an image format, and a sound format. The 
multiplex message forms at least a portion of a 
5 multimedia document and includes object files, each 

object file being represented by at least one data slice. 
The method includes providing an object number counter in 
the data file indicating the number of object files 
following the object number counter in the data file. The 

10 method further includes providing object descriptions, 

each object description describing a corresponding one of 
the object files. The method also includes providing a 
choreography group including the data slices of the 
object files interleaved in a predetermined manner. 

15 The details of the preferred embodiment of the 

present invention are set forth in the accompanying 
drawings and the description below. Once the details of 
the invention are known, numerous additional innovations 
and changes will become obvious to one skilled in the 

20 art. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

FIGURE 1 is a block diagram showing a system 
architecture of the present invention. 

FIGURE 2 shows the overall hierarchy of the 
5 layered parts of a document, including the hierarchy of a 
framed image. 

FIGURE 3 shows an overall display of a multimedia 
document formed according to the hierarchy of FIGURE 2. 

FIGURE 4 is a flow diagram showing the steps by 
10 which a code segment receives image information. 

FIGURE 5 shows a file format of the present 
invention . 

FIGURE 6 . shows a multiplexed scheme in accordance 
with the invention. 

15 FIGURE 7 shows an example of choreography of a 

document. 

FIGURE 8 shows the default method for organizing a 
document of the present invention. 

Like reference numbers and designations in the 
20 various drawings indicate like elements. 
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DETAILED DESCRIPTION OF THE INVENTION 

Throughout this description, the preferred 
embodiment and examples shown should be considered as 
exemplars, rather than as limitations on the present 
5 invention. 

I. System Architecture 

The overall system architecture 100 of the 
invention is shown in FIGURE 1. A bit stream 102 is input 
to the system 100. The input bit stream 102 represents a 
10 new special data format that is referenced in this 

specification as the "ART Format." The ART Format is an 
encapsulated data format that includes various types of 
data . 

The bit stream 102 is input from an overall 

15 information provider 104, for example, an Internet 

provider such as America OnLine®, to the input function 
of a player wrapper 106 as the bit stream 102 is 
received. The player wrapper includes a player dynamic 
link library (DLL) 108. The player DLL 108 separates and 

20 commands play of the various information in the bit 

stream 102. The playing is carried out using a technique 
known as "ART DOC Extensions, " which is also embodied as 
a DLL. Although the playing systems are described as 
being embodied as DLLs, the playing systems alternatively 

25 could be embodied as dedicated hardware components, e.g., 
digital signal processors. Other examples of the player 
wrapper 106 include a Netscape or ActiveX plug-in. 
Player wrappers 106 can also allow editing and creation 
of multimedia documents. 

30 A module of the player wrapper 106 may use a plug- 

in, rich text format text processor. The preferred plug- 
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in text processor is Paige™, commercially available from 
Datapak, Inc. Paige™ carries out many different 
operations and allows other modules to tell Paige® what 
operations to perform what ways. Paige™ can respond to 
5 commands to use various fonts, place text. within a 

multimedia document, and reserve other places within the 
document for other items, such as image data or video. It 
will be understood, of course, that alternate software 
packages and systems could be used. 

10 The Paige™ text processor may perform the 

displaying of the document. In such a case, Paige™ 
communicates with a currently selected window on the 
computer display. 

The preferred framework of the present invention 

15 includes various overheads, such as clipboard, command 
stacks for undo, and other well-known overhead systems. 

II. Construction and Features of an ARTDOC Document 

The multimedia document is ordered as a series of 
layered parts. FIGURE 2 shows the overall hierarchy 200 

20 of the layered parts of a document (referred to herein as 
an "ARTDOC document") in accordance with the present 
invention. The highest layer of the ARTDOC document is 
the document object 202. The document object 202 controls 
information for the entire active display window, 

25 including commands for background colors and enclosing 
windows . 

Each document portion has a capability of 
supporting "children," which are subspecies of the ARTDOC 
document. The document object 202 shown in FIGURE 2 
30 supports two children-a main text object 204 and a frame 
object 206. The main text obiec' 204 can include other 
children, including separator zrhiidren 207, hypertext 
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children 208, and in-line children 210. The hypertext 
children 208 can include links to other documents, which 
can be ARTDOC document images or any other document. The 
in-line children 210 can include in-line bit maps 228, 
5 sounds 230, or other multimedia objects, which can be 
displayed as part of the ARTDOC document. The separator 
children 207 may provide visual lines to separate text. 

The frame 206, as shown, also includes a number of 
children. For example, the frame 206 includes an image 

10 214 within the frame 206, hyperlinks 216 regarding the 
image 214, a sound file 219, and a slide show 217. The 
framed image 206 may also include caption text 218, which 
may also include text within a frame 222, hyperlinks 224, 
and in-line multimedia objects 226 like the main text 

15 object 204. 

FIGURE 3 shows an overall display for an ARTDOC 
document 300 formed according to the hierarchy of FIGURE 
2. A number of objects may be embedded within the 
document 300, along with text 310. The Paige™ word 

20 processing program, as part of its function, controls 
wrapping of lines, etc., around the images displayed in 
the document 300. Paige™, as now commercially available, 
allows its areas to be set-up and controlled. Paige™ also 
allows certain areas to be defined that are not Paige™ 

25 areas, for example, an image and/ or its caption. 

The system according to the present invention uses 
these Paige™ capabilities to create an exclusionary area 
302 within the document 300. A frame object 304, in this 
case including an image 305, is located within the 

30 exclusionary area 302, which effectively forms an 

invisible frame around the frame object 304. The enclosed 
objects can include framed images, slide shows, framed 
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text, sounds, separators, and hyperlinks, which are all 
described in detail below. 

A similar operation is carried out for the 
captioning operation, by which a caption 306 is placed 
5 adjacent the object 304, but within the exclusionary area 
302. In this example, a hyperlink 308 is positioned over 
the graphical image 305. The hyperlink 308 may be an area 
of the image object 305 that one can select, for example, 
by a mouse click, to determine something else about the 

10 image object 305. 

The hierarchical structure 200 of FIGURE 2 
facilitates moving positions of objects within the ARTDOC 
document. Each object within the ARTDOC document is 
formed of a number of objects and sub-objects. The sub- 

15 objects are children of the objects. The hierarchical 
system 200 according to the present invention, as 
described above, allows a higher level object to be moved 
together with all of its lower level objects (i.e., sub- 
objects) . In the example of FIGURE 2, frame 206, image 

20 214, hyperlink 216, and caption text 218 (with its 
children, text 222, hyperlinks 224, and in-line 
multimedia objects 226) are moved together with the frame 
206 when it is moved. Therefore, for example, moving a 
frame will move not only the image within that frame, but 

25 also the hyperlinks and text within the frame. This same 
operation can be used with a delete command. Deleting the 
frame object deletes the frame, the image, the hypertext, 
and all of the other associated parts. 

Each action (e.g., a move command) is passed to 

30 the object, and the .object decides what to do with that 
action. Thus, for example, to move a frame 206, a mouse 
operation is used to move the position of the frame 206 
from a first position to a second position. The mouse 
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operation is handled in the standard way by the operating 
system, which hands off the new position to the player 
106. The player 106 then passes this new position to the 
core DLL 108, which passes it to the object. Now, the 
5 object knows not only its new position, but also the new 
relative positions of all the sub-objects within the 
object . 

The ARTDOC document supports a number of different 
objects. In general, three types of objects exist: those 

10 with a visual representation, those with an audio 

representation, and those with specialized functions. 
Examples of objects with visual representation include 
images, which are generally static bitmaps. Objects with 
audio representation include digitally sampled sounds, 

15 which may be WAV or AIFF files, and MIDI files, which are 
usually played through an FM synthesizer. Specialized 
objects include frame objects, separator objects, and 
hyperlink objects. Frame objects hold multimedia 
objects, such as images, sampled sounds, and MIDI files, 

20 and also may provide a visual representation of the 

document. Separator objects provide a visual horizontal 
line (of different styles) that either separate 
paragraphs of text (moving with the text as it is 
reformed) , or may exist at a geometrical position in the 

25 document. Hyperlink objects allow navigation to other 
content described by the author of the document. The 
present invention also supports ART file objects, which 
are described in copending U.S. Patent Application Serial 
No. 08/755,586, assigned to the assignee of the present 

30 application. ART file objects include slide shows, sound 
and picture objects, ART sounds, and ART images. Any type 
of document can be contained inside an ARTDOC document by 
creating an ARTDOC extension module for the object. 
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Each object may include some kind of command for 
perception. Typically the perception of the object will 
be displaying of video or listening to sound. The command 
for perception will be a command indicating the kind of 
5 player that is used to produce the perception. If the 
player is not resident in the playing computer, various 
steps are taken as described herein. 

In addition, each object may be able to pass and 
receive messages and to supply and retrieve its data. As 

10 an example of these functions, suppose the author wants 
to create an ARTDOC document that includes a video object 
(which includes image and audio portions) from a third 
party library. The library for the video object is 
"wrapped" with an ARTDOC document module, effectively 

15 creating an ARTDOC extension. This "wrapper" module 
allows the ARTDOC system to communicate with the third 
party library so that it can become part of the ARTDOC 
document. The wrapper module provides the translation 
layer and interprets and provides messaging between the 

20 ARTDOC document and the third party library. Referring 
to FIGURE 1, as the data stream 102 is received, the data 
stream meant for the video object is extracted and passed 
to the ARTDOC video object that wraps the third party 
library. The wrapper module then passes the data to the 

25 library, which interprets the data. The ARTDOC document 
cannot interpret the data, because it only carries the 
data. The library then interprets the data and provides, 
for example, a new video image and sound bite back to the 
wrapper module. The wrapper module, in turn, communicates 

30 back to the ARTDOC document, which displays the image at 
the proper place in the document. The video object may 
communicate with sound hardware directly to provide the 
sound. The system may work in this manner for all 
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objects, including the text that is passed to the rich 
text processor and the data that is passed to the player 
106. 

When authoring an ARTDOC document, the user may 
5 produce some text using the Paige™ system. Paige™ will 
place the text on the computer display according to the 
desired type of rich text characteristics. Now, if the 
user wants to place an image within the ARTDOC document, 
the user employs a mouse, for example, to drag and drop 

10 the image into the ARTDOC document. The computer 
operating system that controls and monitors mouse 
operations passes the new information indicative of the 
mouse information (e.g., the drag and drop information) 
into the document 202 shown in FIGURE 2. 

15 The document 202 forms a code segment that 

receives image information according to the flow diagram 
of FIGURE 4, which constructs the particular frame 206. 
The process 400 begins at step 402, where an image, e.g., 
a GIF image, is dropped into an ARTDOC document. The 

20 operating system sends information about the image to the 
player wrapper 106, which notifies the ARTDOC document at 
step 404. The information itself is then passed to a 
decoder at step 406. The decoder determines the data 
format (here GIF) and provides size information and 

25 visual representation back to the ARTDOC document at step 
408. The ARTDOC document encloses the image in a frame 
and adopts the component object as a new child in step 
410. The position of the ARTDOC document may be given by 
the original operating system mouse location. 

30 An important feature of the present invention is 

that the objects forming the images may be generic. Thus, 
ail objects have the same characteristics and can be 
connected together to form parts of the ARTDOC document. 
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The preferred items forming the objects include a 
framed image, a slide show, framed text, sound, a 
separator, and a hyperlink. Each of these will now be 
described in detail. Many of these objects may be 
5 provided by the player 10 6 as .an ARTDOC document 
extension . 

Frame objects (e.g., frame 206) are containers 
that perform several tasks in an ARTDOC document. First, 
and most significantly, they define the exclusionary 

10 areas within the page around which the text will wrap. 
Second, they contain the multimedia types described 
above, i.e., images, sounds, video, etc., or text 
objects, thereby creating framed text objects. Each of 
these multimedia types and text objects may also have 

15 further children, as described above. Finally, frame 
objects may provide a visual representation as an 
enclosing border at the author's option. Examples of 
visual representation options are color, thickness, pen 
style, outer margins, and three-dimensional effects, 

20 which can be fully implemented by the ARTDOC document. 

Image objects (e.g., image 214) have an original 
size and provide a static visual representation. The 
author may override the original size. Image objects may 
be contained in frames or may appear in-line in a block 

25 of text and may be provided by the ART player 106 as an 
ARTDOC document extension. 

Sound objects do not have a size, but are given 
the appearance of a sound icon if they are authored 
without a visual component. Sound objects may be 

30 contained in frames (e.g., sound file 219) or appear in- 
line in a block of text (e.g., in-line sound file 230). 
Sound objects may be of the sample wave type or of the 
sequenced MIDI type. Sound objects may be played by 
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clicking on the visual representation or by other 
direction of the user and may also be provided by the ART 
player 106 as an ARTDOC document extension. 

Sound and picture objects are a combination of a 
5 sound component and an image component, giving the visual 
representation of an image, but allowing the sound to be 
played when clicked. Like sound and image objects, sound 
and picture objects may be provided by the ART player 106 
as an ARTDOC document extension. 

10 Non-static multimedia objects include slideshows 

(e.g., slideshow 217) and video, as well as any similar 
moving-image-with-sound type of representation. Non- 
static multimedia objects are displayed as an image 
(optionally with a title page, if supported) until the 

15 user clicks or directs the object to be played, which 
results in a changing image, usually accompanied by 
sound. These objects may also be provided by the ART 
player 106 as an ARTDOC document extension. 

Textual objects may appear in a frame (sometimes 

20 referred to as framed text objects, e.g., text 222); they 
may also appear in the form of the main body text (e.g., 
main text 204) and as captions for images (e.g., caption 
text 218) . Text objects flow within the boundaries of the 
enclosing object and are stored in the file as an RTF 

25 data stream with embedded hyperlinks. Textual objects are 
implemented as an ARTDOC wrapper around the Paige© text 
processor . 

Separator objects (e.g., hypertext 208) are used 
to provide a visual separation between paragraphs of text 
30 or other objects. Separator objects allow all the visual 
representation options as do frames, but do not contain 
objects. Separators are often used with "Anchor" 
geometry, which will be described below. Separator 
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objects may be fully implemented by the ARTDOC document 
itself . 

Finally, hyperlink objects (e.g., hypertext 208) 
are associated with any of the multimedia object types 
5 (images, sounds, slideshows, etc.) or with textual 
objects. When associated with multimedia objects, the 
hyperlink is described as a geometric (preferably 
rectangular) region on the associated object. When 
associated with text, the text appears visually different 

10 according to the author's specification. The user 

receives feedback when moving the mouse over a hyperlink 
region or hyperlink text, which includes a description of 
the location to which the hyperlink points. When the 
user clicks on the region or text, the appropriate 

15 information is retrieved and displayed for the user, or 
the user is taken to the location designated by the 
hyperlink pointer. Hyperlink objects are also fully 
implemented within the ARTDOC document. 

The architecture of the ARTDOC document of the 

20 present invention is fully extensible, meaning that the 
document will support any kind of multimedia object. 
This includes image, sound, video, and text stream 
object. With any multimedia object, the data for that 
object will be delivered and progressively displayed or 

25 played. 

The "geometry" of an object refers to the 
specification for the dynamic positioning of the object 
within the bound of its parent- or controlling object. Any 
visual object in an ARTDOC document has four edges: left, 
30 top, right, and bottom. The boundaries of such an object 
can be fully determined by applying a single "rod," 
"spring," '"measure," or "anchor" to each edge. A rod is a 
fixed delta in parent coordinates from an edge of a child 



WO 98/54637 



PCT/US98/11060 



- 19 1 

object to the same edge of its parent object. A spring is 
a percentage of the parent's total. width or height from 
which the position of the child's edge should be 
determined. Springs calculate from the center of the 
5 object when paired with a measure; otherwise, the 

position of each edge is calculated independently. An 
anchor is a fixed delta in parent coordinates from the 
bottom edge of the first character of a paragraph to the 
top edge of a child object. A measure is the length of an 

10 edge (width or height) of an object and is used to fix 
the width or height. 

The most common geometry is that of a fixed- 
position object that is described using a rod for the top 
and left sides of the object and a measure from the 

15 bottom and right sides. An object with this geometry 

retains its relative position form the top, left corner 
of the controlling object (in this example, the parent 
document) and also does not change size when the document 
is resized. An object with horizontal centering geometry 

20 is created by using a spring (set at 50%) for the left 
edge and a measure for the right edge. This causes the 
position of the object to move as the document resizes 
(remaining in the center), but still does not' change the 
size of the object. 

25 Other applications may include a geometry that 

attaches an object to the right edge of a document by 
using a left-edge measure and a right-edge rod. Allowing 
an object to grow with the document is also supported by 
attaching rods to both sides and not using measures, thus 

30 removing the fixed-measure constraint. 

The anchor geometry setting allows an object's 
vertical position to be dependent or. the position of a 
block of text. This setting is t.c-st. often used with 
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separators to ensure that they remain between the blocks 
of text that they separate, but can be used with any 
object. 

III. The ARTDOC Document Pile Format 

5 FIGURE 5 shows the preferred ARTDOC document file 

format 500 of the present invention, which includes a 
header area 502, an object archive 504, and a multiplex 
section 516. The header area 502 includes typical header 
information, such as start bits and version number, and 

10 any other overhead information. 

The object archive 504 follows the header area 502 
in the file format 500. The object archive 504 stores 
object information, but does not store the data 
associated with that information. The object archive 504 

15 includes the information from the hierarchy of FIGURE 2. 
Each element in the hierarchy 200, from top to bottom, is 
used to provide an explanation of the information about 
each element of the hierarchy and its children. In the 
preferred embodiment of the present invention, the 

20 document is traversed in a "depth-first" manner, meaning 
that the layout and attributes of each element of the 
document are provided in the object archive 504. 
Consequently, the object archive will include information 
about the geometry (layout, position, and size, as 

25 described above) of the document, as well as complete 
descriptions of the document attributes, such as the 
frame . 

The object archive 504 is the first information 
received by a receiving element. Hence, the receiver can 
30 display layout information about an ARTDOC document as 

soon as the receiver has received the object archive 504. 



WO 98/54637 



PCT/US98/U060 



- 21 - 

As explained above, however, the data to fill that 
layout is not received with the object archive 504. 

The invention may have several features and 
requirements. First, the invention may allow a first 
5 portion of an object's data to be a splash or "miniature 
representative rendering" of the object. Whether an image 
has a splash is determined by the particular format of 
the object in question. The splash data, if present, 
appears at the front of the image stream in order to 

10 provide the splash early in the download. The splash 
image indicates some very coarse information about the 
content of the multimedia objects in the ARTDOC document. 
The splash information is progressively updated as more 
data arrives that describes the images. More items of 

15 image information can be provided to the object to 
display more information. Text can also arrive and be 
displayed. . 

The data for each of the multimedia objects "(e.g., 
images, sounds, text streams, or video) is delivered in 

20 the multiplex section 516 of the ARTDOC file 500. The 

data is generally the same data that would be resident in 
a file containing the object alone, without any of the 
geometry and attribute information, as this information 
is contained in the object archive 504. The multiplex 

25 section 516 represents the bulk of the data stream. 

In the multiplex section 516, data is interleaved 
together according to the "choreography" of the document. 
The term choreography, as used herein, refers to the 
ordering of the multimedia objects in the ARTDOC file 

30 format. When objects, such as images, are designated to 
appear during the same time interval, their data is 
interleaved together in the multiplex section 516. As the 
ARTDOC file 500 is received, the interleaved data stream 
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is decomposed, and the appropriate data is delivered to 
each object, progressively updating each object's visual 
appearance in the display of the ARTDOC document. 

Some classes of objects, however, are not 
5 interleaved with other objects in the multiplex section 
516, regardless of the author's choreography designation. 
The first class includes those objects that cannot be 
progressively rendered. MIDI files and most standard 
types of audio and video files are examples of this class 

10 of objects. Because these objects must be completely 
downloaded before they can be interpreted, no benefit 
flows from interleaving their data. The second class of 
non-interleaved objects includes files that are designed 
to be played back progressively, but that are authored 

15 for a particular bandwidth. These objects are referred to 
as time-based or "temporal" files. Examples of such 
temporal files are certain audio and slide show files 
that cannot be interleaved with other data without 
slowing the delivery of their own temporal data and 

20 risking starving their players of data. 

The classes of objects that are not interleaved 
are detected by the ART DOC document and are placed in 
their own choreography groups within the multiplex 
section 516. These groups may be arranged with 

25 interleaved multiplex groups without degrading document 
playback, as interleaved and non-interleaved groups do 
not overlap. 

By interleaving certain data, such as images, and 
separating temporal files, the present invention permits 
30 the incremental "display" of the document (including 
sounds and images) to the user". As described above, 
images can be multiplexed or mixed together so that they 
arrive and are displayed at substantially the same time. 
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The constraints of narrow-band, i.e., modem 
communication, however, generally prevent interleaving of 
the temporal objects; the quality of their playback would 
likely suffer if they were arbitrarily mixed together. 
5 None of the proposed encapsulated formats for HTML allows 
such an incremental display and do not allow for the 
proper organizing of temporal information for narrow-band 
applications. The proposed HTML formats rely on broad 
band communication, meaning that these formats may 

10 arbitrarily mix the data, regardless of the type of data, 
and may sequentially store objects, sacrificing either 
the audio or video presentation of the HTML document. 

It is known that particular object data, such as 
text streams, may not be compressed. The ARTDOC document 

15 can compress such data itself into the multiplex section 
516. If this is done, the compression flag (see reference 
numeral 708 of FIGURE 7) is set. 

An unknown object is a special type of object 
within the ARTDOC format. The unknown object is an object 

20 that probably has a defined player, but the computer 
receiving the unknown object does not recognize the 
defined player. The unknown object includes the 
information contained in the object archive 504 
(described above) , because the unknown object is a type 

25 of object. Therefore, the receiving computer knows where 
this information is and how large it is. Hence, the 
player 106 can draw the outlines of the unknown object, 
without putting the particular details in it. At least 
some of the data from the multiplex section 516 is 

30 provided to the unknown object, which displays an 

"unknown" icon. Although that data cannot be displayed, 
it can be carried around by the file embodying the 
unknown object, and the file can be saved intact. 
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Moreover, later downloads of coded information can 
be made to allow playing of the unknown object. For 
example, the unknown object may reference a DLL of which 
the playing system 106 has no knowledge. Later download 
5 of the DLL may allow later display of the unknown object. 

The ability to support unknown object types also 
allows the use of user-defined object types. So long as 
the object type follows the conventions given above, it 
can be made into part of the ARTDOC format, and 
10 encapsulated within the ARTDOC document. 



IV. The Multiplexed Scheme 

The preferred embodiment of the present ' invention 
uses a multiplexed scheme, as noted with respect to 
FIGURE 5. The multiplexed scheme preferably employs 

15 "slices" of information. Each slice represents a piece of 
information that can make some degree of difference in 
the perceived ARTDOC document. A slice may be a segment 
of audio between any two natural pauses in the audio, a 
piece of an image that causes a further update of the 

20 image, or a block of text. 

FIGURE 6 shows the preferred multiplex message 516 
in accordance with the multiplexed scheme of the present 
invention. The multiplex message 516 includes an object 
number counter 602. The object number counter 602 

25 includes an indication of the number of "objects" that 
will follow in the object section of the multiplexed 
information . 

Object descriptions 604 follow the object counter 
602. The object descript ions 604, being of the same 
30 number as the object number counter 602, include an 

object reference 606 that provides a back reference to 
the object information stored in the object archive 504 
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and a flag 608 indicating whether the object data was 
compressed by the ARTDOC document. 

Following the object descriptions 604 are 
choreography groups 610. There may be any number of 
5 choreography groups 610 (as described below) 

corresponding to the ordering of interleaved data within 
the document. When no further groups exist in the 
multiplex section 516, a single zero-byte 624 is used to 
mark the end of the multiplex section 516."- 

10 A choreography group 610 includes an object 

counter 612 that indicates the number of objects 
contained in the particular choreography group 610. 
Objects may appear in different groups, providing a 
portion of their data from within each group. Following 

15 the object counter 612 is a section 614 providing the 
size and type information for each object in the 
choreography group 610. The size and type section 614 
provides an internal characterization of the multiplex 
data as well as the total length of the multiplex data 

20 for the object contained in this particular choreography 
group 610. An object header information section 616 
follows the size and type information section 614. The 
object header section 616 includes a section 618 that 
contains the length of the object's data slice within the 

25 group 610 and an object reference section 620 that 

provides a back reference to the object's description 
604. The data slice length section 618 is an optional 
parameter, because, if only one object is contained in 
the group 610 (i.e., the object counter 612 indicates 

30 "1"), the slice length is assumed to be the same as the 
length information provided in size and type section 614. 

The actual interleaved data 622 follows the object 
header information 616. The interleaved data 622 
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constitutes the bulk of the information included in the 
choreography group 610 and includes the data destined for 
the referenced object. The interleaved slice data 622 is 
preferably provided as a series of raw data bytes. The 
5 length of the interleaved slice data is given by the data 
slice length section 618 for each object in the 
choreography group 610. Each object supplies its 
indicated length of data in turn, repeating until 
completion of the full amount of data given by the size 

10 and type section 614 for each object. The last slice of a 
particular object may be shorter than the size given by 
the size and type section 614. In such a case, the last 
slice will correspond to the remaining total bytes to be 
delivered in the group 610, as indicated by the size and 

15 type section 614. Reference numeral 626 is an exemplary 
interleaved slice data section for two objects. If the 
object count 612 indicates "1", there is only one object 
in the group 610 and only one slice of data of the length 
given by the single entry in the size and type section 

20 614. These single object groups represent non-interleaved 
data, as described above. 

The author may place objects or portions of 
objects into the choreography groups 610 by means of the 
user-interface. By placing an object into the same 

25 choreography group as another object, the author directs 
that those objects will be delivered interleaved 
together. Consequently, those interleaved objects will 
appear progressively over the same duration during the 
file download. Objects placed in later choreography 

30 groups will appear in time after objects placed in 

earlier groups. The data for a particular object may be 
placed in different choreography groups to create, for 
example, the effect of having only the initial splashes 
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of images appear early in the document, followed by the 
text of the document in a later group, and finally by the 
remainder of the image data in even later groups. 

An example of choreography for an ART DOC document 
5 700 is shown in FIGURE 7, which illustrates three 
displays of the document 700 over time. In this 
embodiment, portions of the entirety of the ARTDOC 
document 700 can be sent simultaneously. A first part of 
the ARTDOC document 700 is displayed at time r r l as a 

10 "splash" of all images 702 and. 703. This is followed by a 
second part of the document 700 at time T 2 , including more 
information about the images 702, 703 as well as text 
part 1 704. Time T, shows the final ARTDOC document 700, 
with completed images 702, 703, as well as additional 

15 text parts 2 and 3, 706 and 708, respectively. 

If the user does not specify a particular 
choreography, a default organization will be provided. 
The default organization preferably groups objects from 
top to bottom as they appear in the ARTDOC document. 

20 Moreover, in the default, objects that are located 

roughly near one another may be placed in the same group 
to ensure that they will be delivered together. 

FIGURE 8 illustrates the default method for 
organizing an ARTDOC document 800. The ARTDOC document 

25 800 is conceptually arranged into a number of areas, 

called "buckets." Each bucket can be a physical area on 
the ARTDOC document 800, or alternatively could be a 
portion of different areas on the ARTDOC document 800. 
FIGURE 8 shows the ARTDOC document 800 being broken up 

30 into four physical sections-Section 1 810, Section 2 820, 
Section 3 830, and Section 4 840. Section 1 810 includes 
both text 812 and an image II 814. Section 2 820 includes 
text 822 and an image 12 824, as well as a portion 826 of 
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image 13. Section 3 830 includes another portion 832 of 
image 13 and some text 834. Section 4 840 includes only 
text 842. 

Each of these sections 810, 820, 830, 840 is 
5 treated as a bucket, and each bucket represents an area 
on the ART DOC document 800. Each bucket is analyzed to 
determine slice sizes that make sense. An appropriate 
slice should be big enough to alter the perception 
presented to the user,, but small enough to enable 

10 simultaneous occurrence of events. 

Bucket 1 (i.e., Section 1 810) has two objects 
including the text portion 812 and image II 814. Bucket 2 
(i.e., Section 2 820) has three objects, the text 822, 
the image 12 824, and the portion 826 of image 13. Bucket 

15 3 (i.e., Section 3 830) has two objects, the other 

portion 832 of image 13 and the text 834, and Bucket 4 
(i.e., Section 4 840) has only one object, the text 842. 
Therefore, the system of the present invention includes 
multiple areas forming the ARTDOC document 800. 

20 The integrated file format of the present 

invention allows all parts of the image to be received in 
sequence. The sequence can be set by the author of the 
document being perceived. The sequence is also such that 
at least part of the sequence allows portions of the 

25 information in the document to be progressively received. 

A number of embodiments of the present invention 
have been described. Nevertheless, it will be understood 
that various modifications may. be made without departing 
from the spirit and scope of the invention. Accordingly, 

30 it is to be understood that the invention is not to be. 
limited by the specific illustrated embodiment, but only 
by the scope of the appended claims. 
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CLAIMS 

WHAT IS CLAIMED IS: 

1. A method for producing. a hierarchical data file 
for a multimedia document, the data file having 

5 different file formats encapsulated within the 

data file, the method comprising: 
a. encapsulating in a multimedia document a 
first file support object including 
information in a first file format; 
10 b. supporting the first file support object by 

the multimedia document; 

c. encapsulating in the multimedia document a 
second file support object including 
information in a second file format different 

15 from the first file format; and 

d. supporting the second file support object by 
the multimedia document. 

2. The method of claim 1, further comprising changing 
at least one of the objects in the data file. 



20 3. 



The method of claim 1,. further comprising adding 
at least one object to the data file. 
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4. The method of claim 1 wherein the data file is 
displayed in a window on a computer display, the 
method further comprising: 

a. creating an exclusionary area within the 
5 window; and 

b. locating an object within the exclusionary 
area, the object being selected from a group 
of data objects including a framed image, a 
slide show, framed text, sound data, a 

10 separator, or a hyperlink. 

5. The method of claim 1 wherein the data file 
includes splash image data defining a splash 
image, the method further comprising locating the 
splash image data within the data file such that 

15 the splash image is displayed on a computer 

display as the splash image data is received by a 
receiver coupled to the computer display. 

6. The method of claim 5, further comprising locating 
update splash data that further defines the splash 

20 image within the data file such that the splash 

image is updated on the computer display as the 
receiver receives the update splash data. 

7. The method of claim 1, further comprising 
providing each object with an address indicating a 

25 player that plays the object. 

8. The method of claim i, further comprising 
compressing the information in each object. 
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9. The method of claim 1 wherein the data file is 
downloaded by a receiving computer, the method 
further comprising: 

a. creating an unknown object in the data file; 
and 

b. locating player data within the unknown 
object defining a player that plays the 
unknown object. 

10. A hierarchical data file structure that 
encapsulates a plurality of different file formats 
to form a multimedia document, the multimedia 
document being capable of being displayed on a 
display of a computer system, the data file 
comprising : 

a. a document including information for 
controlling the display; 

b. a first support object including information 
in a first file format, the first support 
object being encapsulated in the document and 
being capable of supporting a plurality of 
first lower objects, each first lower object 
being a lower level object than the first 
support object in the hierarchical data file 
structure; and 

c. a second support object including information 
in a second file format different from the 
first file format, the second support object 
being encapsulated in the document and being 
capable of supporting a plurality of second 
lower objects, each second lower object being 
a lower level object than the second support 
object in the hierarchical data file 
structure . 
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11. The hierarchical data file structure of claim 10 
wherein the first file format is selected from a 
group of file formats including a textual file 
format, an image file format, and a sound file 
format; and wherein the second file format is 
selected from a group of file formats including a 
textual file format, an image file format, and a 
sound file format. 

12. The hierarchical data file structure of claim 10 
wherein, when an operation is performed on a 
higher level object of the hierarchical data file 
structure, the operation is also performed on all 
objects occupying a lower level in the hierarchy 
than the higher level object. 



15 13. The hierarchical data file structure of claim 10 
wherein each object has a plurality of common 
attributes, including a command for perception of 
the object, an ability to pass and receive a 
message, and an ability to supply and retrieve the 

20 data embodied in the object. 

14. The hierarchical data file structure of claim 10 
wherein each object is a generic element of the 
hierarchical data file structure, such that any 
combination of objects can be grouped together to 
25 form a part of the multimedia document. 
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15. The hierarchical data file structure of claim 10 
wherein the document forms a code segment that 
receives image information; and wherein the image 
information is used to construct an image frame 
for a framed image that is part of the multimedia 
document . 

16. The hierarchical data file structure of claim 15 
wherein the framed image has an image data format; 
and wherein a decoder determines the image data 
format and encapsulates the framed image with the 
image frame. 

17. A file format for storing a plurality of different 
types of data so that the plurality of data types 
can be displayed on a computer display as a 
multimedia document, the multimedia document 
including a plurality of multimedia object files, 
the multimedia document being embodied in a 
hierarchical data file, each object file having a 
level in the hierarchy of the data file, the file 
format comprising: 

a . a header; 

b. an object archive for storing information 
about the plurality of object files, 
including information about the level of each 
multimedia object file within the hierarchy; 
and 

c. a multiplex section including data for each 
of the multimedia object files of the 
multimedia document. 
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18. The file format of claim 17 wherein the multimedia 
object files in the multiplex section are each 
played by a player as the multiplex object file is 
received by a receiver. 

5 19. The file format of claim 17 wherein the data for 
at least some of the multimedia object files is 
interleaved. 



The file format of claim 17 wherein the object 
archive includes data defining geometry of the 
multimedia document. 

The file format of claim 17 wherein each of the 
multimedia object files is defined by at least one 
data slice; and wherein the multiplex section 
further includes: 

a. an object number counter indicating the 
number of multimedia object files following 
the object number counter; 

b. a plurality of object descriptions, each 
object description corresponding to one of 
the multimedia object files, each object 
description including an object reference 
that refers to the corresponding object 
information in the object archive and a flag 
indicating whether the corresponding object 
data is compressed; and 

c. a choreography group providing information 
about a first group of multimedia object 
files . 



WO 98/54637 



PCT/US98/11060 



- 35 - 

22. The file format of claim 21 wherein each 
multimedia object file is divided into at least 
one data slice and the choreography group 
includes : 

5 a . an object counter indicating the number of 

multimedia object files in the choreography 
group; 

b. size and type data for each multimedia object 
file; 

10 c. header data; and 

d. the data slices of the multimedia object 
files interleaved together. 

23. The file format of claim 17, further comprising a 
non-multiplex section following the multiplex 

15 section, the non-multiplex section including a 

plurality of separate object files that are not 
played by a player as the separate object files 
are received by a receiver. 

24. A method of encoding a framed image in a frame, 
20 the frame to be included as part of a multimedia 

document, the multimedia document encapsulating 
data in a plurality of file formats and being 
capable of being displayed on a display of a 
computer, the computer including a player, 
25 comprising: 

a. placing an image into the multimedia 
document ; 

b. receiving image information about the image 
by the player; 

30 c. decoding the image information; 

d. sending the decoded image information to the 
multimedia document; and 
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e. enclosing the decoded image information into 
a frame in the multimedia document. 

25. A method for multiplexing data in a multiplex 
message that includes data in a plurality of file 
formats, the file formats being selected from a 
group of file formats including a textual file 
format, an image file format, and a sound file 
format, the multiplex message forming at least a 
portion of a multimedia document and including a 
plurality of object files, each object file being 
represented by at least one data slice, the method 
comprising : 

a. providing an object number counter in the 
data file indicating the number of object 
files following the object number counter in 
the data file; 

b. providing a plurality of object descriptions, 
each object description describing a 
corresponding one of the object files; and 

c. providing a choreography group including the 
data slices of the object files interleaved 
in a predetermined manner. 

26. The method of claim 25, further comprising 
providing a first player pointer including an 
address of a player that plays the choreography 
group . 
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27. The method of claim 25 further comprising locating 

a plurality of slice size data blocks before the 
interleaved data slices, - each slice size data 
block corresponding to one of the data slices and 
providing a size of the corresponding data slice. 



28. A file format for storing a plurality of different 

types of data so that the plurality of data types 
can be displayed on a computer display as a 
multimedia document, the multimedia document 
including a plurality of multimedia object files, 
the multimedia document being embodied in a 
hierarchical data file, each object file having a 
level in the hierarchy of the data file, the file 
format comprising an object archive for storing 
information about the plurality of object files, 
including information about the level of each 
multimedia object file within the hierarchy. 



29. A file format for storing a plurality of different 

types of data so that the plurality of data types 
can be displayed on a computer display as a 
multimedia document, the multimedia document 
including a plurality of multimedia object files, 
the file format comprising a multiplex section . 
including data for each of the multimedia object 
files of the multimedia document. 



30. 



The file format of claim 29 wherein at least some 
of the data in the multiplex section corresponding 
to different object files is interleaved. 
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